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END  STATE  -  COMMANDER’S  VISUALIZATION  AT  THE  COMPANY  LEVEL:  TRAINING 
REFINEMENT  AND  TRANSITION 

EXECUTIVE  SUMMARY 


Research  Requirement: 

Success  in  modem  military  operations  depends  on  the  ability  of  commanders  to  visualize  an 
increasingly  complex  and  unpredictable  operational  environment.  Visualization  is  the  process  of 
developing  situational  understanding  and  envisioning  how  to  move  the  force  from  its  current  state  to 
the  desired  end  state.  It  is  a  critical  command  skill  that  must  be  acquired  earlier  in  a  leader’s  career  than 
ever  before  given  the  current  operational  challenges  that  small  units  confront.  However,  junior  leaders 
must  all  too  often  develop  their  visualization  skills  through  a  trial  and  error  process  during  collective 
training.  Training  is  needed  that  will  improve  commander’s  visualization  through  deliberate  reflection 
and  practice  coupled  with  performance  assessment  and  expert  feedback  and  guidance. 

Procedure: 

To  meet  this  requirement,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences  researched  and  developed  a  training  product  called  END  STATE  -  Commander’s 
Visualization  at  the  Company  Level.  This  report  focuses  on  follow-on  research  conducted  to  refine  and 
transition  the  END  STATE  training  product  to  the  U.S.  Army’s  junior  leaders.  The  research  addressed 
the  following  objectives:  improve  the  set  of  74  learning  assessments  embedded  throughout  END 
STATE’S  14  training  vignettes;  improve  access  to,  and  the  utility  of,  learner  performance  data  as  well  as 
expert  guidance  and  feedback;  extend  END  STATE’S  potential  training  audience  to  include  both 
individuals  and  groups;  and  refine  and  evaluate  END  STATE’S  pre-  and  post-training  tests  to  ensure 
their  comparability  for  conducting  training  effectiveness  evaluation.  To  meet  these  objectives,  the 
research  involved  iterative  reviews  of  training  and  test  materials  by  active  duty  and  recently  retired 
subject  matter  experts  to  refine  the  relevance  and  generality  of  END  STATE’S  principles  and  practices 
for  visualizing  small  unit  operations  in  counterinsurgency  (COIN)  environments.  Product  revisions 
were  made  to  support  on-line  collection  of  learner  performance  data  and  comments.  Additional  product 
refinements  broadened  END  STATE  from  a  lock-step  individual  trainer  to  an  open-ended  training  tool 
for  small  group  professional  development.  Test  evaluations  with  over  150  junior  leaders  including 
captains,  lieutenants,  senior  non-commissioned  officers  (NCO),  and  cadets  assessed  the  tests’ 
comparability  and  ability  to  discriminate  the  visualization  skills  of  inexperienced  versus  experienced 
junior  leaders. 

Findings: 

Refinements  to  END  STATE’S  learner  assessments  should  substantially  enhance  their 
effectiveness  as  thought  prompts  and  diagnostic  indicators.  Extensive  revisions  were  made  by  subject 
matter  experts  in  response  to  junior  leader  recommendations  for  improving  END  STATE  and  recent 
doctrinal  guidance  on  counterinsurgency.  The  refinements  made  to  improve  the  availability  and  utility 
of  expert  guidance  will  help  ensure  END  STATE  provides  relevant  and  timely  feedback  to  learners  as 
well  as  training  management  data  to  trainers.  The  refinements  made  to  performance  assessment  enable 
on-line  collection,  storage,  and  dissemination  of  learner  performance  data  and  comments.  By  including 
an  “un-locked”  trainer  profile  in  END  STATE,  the  product  can  now  be  used  in  group  settings  to 


v 


stimulate  and  guide  leader  development  and  professional  discussion.  Refinement  and  subsequent 
evaluation  of  END  STATE’S  pre-  and  post-tests  found  that  the  tests  are  sufficiently  comparable  for 
training  effectiveness  evaluation  and  able  to  discriminate  significant  differences  in  the  visualization 
skills  of  junior  leaders  with,  versus  without,  deployment  experience. 

Utilization  and  Dissemination  of  Findings: 

During  prior  formative  evaluations,  48  captains,  lieutenants,  and  senior  NCOs  concluded  that 
the  END  STATE  training  product  is  relevant,  effective,  and  worth  using.  Given  the  additional 
refinements  and  findings  reported  here,  the  authors  conclude  that  END  STATE  is  a  relevant  learning 
tool  that  will  help  junior  officers  and  NCO  leaders  develop  the  visualization  skills  needed  to  understand 
and  adapt  to  the  challenges  of  today’s  COIN  environments.  The  product’s  centralized  storage  of 
training  data  supports  Army-wide  training  management  and  training  effectiveness  evaluation.  END 
STATE’S  potential  training  audiences  include  all  junior  leaders,  both  officers  and  NCOs,  across  a  broad 
range  of  training  venues.  The  product’s  DVD  format  readily  accommodates  training  delivery  options 
ranging  from  a  lieutenant  with  a  laptop  to  a  group  in  a  battalion-level  NCO  professional  development 
(NCOPD)  session.  END  STATE’S  self-paced  design  and  instructorless  expert  feedback  is  a  powerful 
tool  for  individual  professional  development.  With  the  addition  of  a  knowledgeable  trainer  and  a 
theater-style  set-up,  END  STATE  becomes  an  exceptional  tool  for  both  unit  professional  development 
and  the  formal  classrooms  of  the  officer  and  NCO  educational  systems. 


vi 


END  STATE  -  COMMANDER'S  VISUALIZATION  AT  THE  COMPANY  LEVEL:  TRAINING 
REFINEMENT  AND  TRANSITION 

CONTENTS 


Page 


Introduction .  1 

END  STATE  Learning  Objectives .  3 

END  STATE  Overview .  4 

Soldier  Summary .  4 

Course  Map .  5 

Training  Support  Materials .  6 

Training  Vignettes  and  Thought  Prompts .  7 

Pre-  and  Post- Tests .  9 

Training  Data  Presentation .  10 

Learner  Data .  10 

Trainer  and  Researcher  Data .  12 

Product  Transition  Status .  12 

Recommended  Future  Directions .  12 

Conclusions .  13 

References .  15 

Appendix  A.  Acronyms . A- 1 

Appendix  B.  Pre-  and  Post-Test  Refinement  and  Evaluation . B-l 

Appendix  C.  Test  Form  Quality  Rating  Survey . C-l 

List  of  Figures 

Figure  1.  Overview  of  END  STATE’S  training  methods  and  design  features .  2 

Figure  2.  Overlapping  friendly  and  enemy  OODA  cycles  with  external  players .  3 

Figure  3.  Soldier  Summary .  5 

Figure  4.  END  STATE’S  Course  Map .  6 

Figure  5.  Sample  feedback  to  a  learner  on  a  thought  prompt .  8 

Figure  6.  Test  performance  by  test  form  and  junior  leader  experience . 10 

Figure  7.  Sample  feedback  to  learner  on  a  completed  training  vignette . 1 1 

Figure  8.  Sample  answer  key  for  a  thought  prompt . 1 1 


viii 


Introduction 


In  today’s  counterinsurgency  (COIN)  environment,  the  Army’s  junior  leaders  conduct  some  of 
the  most  complex  and  challenging  small  unit  operations  in  military  history.  As  always,  the  ability  to 
visualize  operations  requires  that  these  junior  leaders  develop  situational  understanding  and  envision 
how  to  move  their  force  from  its  current  state  to  the  desired  end  state.  However,  the  challenges  and 
“costs”  of  visualizing  are  severely  compounded  in  COIN  environments.  The  challenges  include 
unpredictable  and  asymmetric  threat  forces,  foreign  people  and  cultures,  dense  and  dysfunctional  urban 
infrastructures,  turbulence  in  unit  personnel,  emerging  technical  threats  and  opportunities,  the 
complexities  of  joint,  interagency,  multinational,  host-nation  operations,  and  particularly,  far  from 
decisive  end  states. 

The  development  of  visualization  expertise,  like  any  other  higher-order  skill,  requires  highly 
structured  and  focused  learning  methods  that  include  repetition,  emphasis  on  challenge  areas,  and 
performance  feedback  (Ericsson,  1996;  Shadrick  &  Lussier,  2004).  A  leader's  ability  to  visualize  is  a 
human  performance  requirement  developed  through  education,  training,  practice,  and  experience 
(Department  of  the  Army  [DA],  2003;  Sanderson  &  Akerley,  2007).  To  improve  commanders' 
visualization  skills,  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI) 
conducted  a  program  to  research  and  develop  visualization  training  for  company  and  battalion 
commanders  (Leedom,  et  al.,  2007;  Lickteig,  Cianciolo,  Silverman,  Menaker  &  Stroupe,  2009; 

Lickteig,  et  ah,  2009;  Shadrick,  Leedom,  Bell,  Manning  &  Lickteig,  2008;  Sidman  &  Garrity  2007). 

END  STATE  -  Commander’s  Visualization  at  the  Company  Level  is  an  instructorless, 
interactive  training  product  to  help  junior  leaders  develop  the  visualization  skills  needed  to  understand 
and  adapt  to  today’s  operational  environments  (Lickteig,  et  ah,  2009).  It  employs  instructional 
strategies  that  ARI  has  successfully  applied  to  developing  leader  cognition  in  a  variety  of  domains 
(Lussier,  Shadrick,  &  Prevou,  2003;  Schaefer,  Shadrick,  Beaubien,  &  Crabb,  2008).  An  overview  of 
END  STATE’S  training  methods  and  design  features  are  provided  in  Ligure  1. 

END  STATE’S  training  vignettes  feature  a  company  commander  and  his  unit  conducting  a  series 
of  missions  in  Kirkuk,  Iraq.  These  vignettes  immerse  learners  in  the  complex  physical  and  human 
terrain  and  the  ambiguous  enemy  situations  that  typify  COIN  environments.  The  product  also  includes 
a  pre-test  and  post-test  to  assess  the  visualization  skills  of  junior  leaders.  In  particular,  END  STATE 
provides  expert  guidance  and  feedback  that  stresses  the  need  to  adapt  solutions  to  the  complexity  and 
uncertainty  of  COIN  and  promotes  the  principles  of  how  to  visualize  small  unit  operations  and  how  the 
principles  apply  in  every  operational  environment. 

The  training  audiences  for  END  STATE  potentially  include  all  junior  leaders,  both  officers  and 
non-commissioned  officers  (NCO),  across  a  broad  range  of  training  and  development  venues.  The 
product’s  client-side,  DVD-based  format  readily  accommodates  training  delivery  options  ranging  from 
a  lieutenant  with  a  laptop  to  a  group  in  a  battalion-level  NCO  professional  development  session.  END 
STATE’S  self-paced  design  and  instructorless  expert  feedback  is  a  powerful  tool  for  individual 
professional  development.  With  a  knowledgeable  trainer  and  a  theater-style  set-up,  END  STATE 
becomes  an  exceptional  tool  for  professional  development  and  the  formal  classrooms  of  the 
officer/NCO  education  system. 
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Training 

■  Interactive  multi-mediaon  DVD 

■  Middle  East  scenarios 

■  Instructorless  coaching 

■  Proactive  role  model 

■  Reflect -act  cycle 

Evaluation 

■  Over  70  checks-on-learning 

■  Multi  pie  choice  with  three  attempts 

■  Objective  and  quantitative  results 

■  Expert  guidance  and  feedback 

■  Pre-  and  post -tests 


Support  i  w 

— *  Quick  Start  Guide - Virion  version  of  End  State 

■  Navigational  prompts  in  blue 

■  Training  status  in  green,  amber  and  red 

■  Introduction,  Road  to  War,  Battle  Update  Briefs,  Maps,  and  Bookmarks 


Figure  1.  Overview  of  END  STATE’S  training  methods  and  design  features. 

During  prior  research  by  ARI  on  the  initially  developed  version  of  END  STATE,  48  junior 
officer  and  NCO  participants  in  a  formative  evaluation  concluded  that  END  STATE  was  relevant, 
effective,  and  worth  using  (Lickteig,  et  al.,  2009).  On  the  strength  of  that  Soldier  support,  ARI 
conducted  follow-on  research  to  refine  and  transition  END  STATE  to  better  prepare  the  Army’s  junior 
leaders.  This  report  documents  that  follow-on  research  that  primarily  addressed  the  following  research 
objectives: 

■  Revise  checks-on-leaming  to  enhance  their  effectiveness  as  thought  prompts  and  as  diagnostic 
indicators  of  performance. 

■  Add  features  that  enhance  END  STATE'S  usability  and  accessibility  to  training  audiences. 

■  Revise  and  evaluate  pre-  and  post-training  tests  to  ensure  test  comparability  for  assessing 
training  effectiveness. 

The  ultimate  goal  of  this  research  was  to  deliver  refinements  that  will  facilitate  END  STATE’S 
transition  to  the  Army  and  improve  the  visualization  skills  of  junior  leaders.  The  purpose  of  this  final 
report  is  to  describe  and  document  the  current  version  of  END  STATE  which  is  available  in  a 
companion  research  product  (ARI,  in  preparation).  This  report  concludes  with  recommendations  for 
conducting  training  effectiveness  evaluation  of  END  STATE  and  for  the  implementation  of  END 
STATE  across  a  broad  range  of  training  venues. 
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END  STATE  Learning  Objectives 


The  Observe-Orient-Decide-Act  (OODA)  cycle  (DA,  2003)  was  adopted  as  the  organizing 
framework  for  END  STATE.  A  series  of  cognitive  task  analyses  (Lickteig,  et  al.,  2009)  focused  on  the 
challenges  of  COIN  (DA,  2006)  and  concluded  that  the  OODA  cycle  provides  a  proactive  framework 
for  visualization  that  enables  and  reinforces  the  rapid  tempo  of  small  unit  operations.  Conceptualizing 
command  decision  making  as  an  OODA  cycle  underscores  that  visualization  integrates  perception, 
cognition,  and  action.  Successful  commanders  repeatedly: 

•  Observe  the  situation  and  proactively  collect  information. 

•  Orient  their  unit  and  external  players  with  situational  understanding. 

•  Decide  what  to  do  and  how  to  do  it. 

•  Act  to  exploit  opportunities  and  deter  threats. 

As  shown  in  the  Figure  2,  END  STATE  stresses  that  the  OODA  cycles  of  the  friendly  force 
“stay  inside”  the  OODA  cycles  of  the  enemy  force.  The  figure  also  reflects  the  complexities  of  COIN, 
particularly  the  need  to  orient  the  many  external  players  that  directly  impact  the  operations  of  small 
units,  such  as  civilians,  tribal  leaders,  host  nation  security  forces,  coalition  forces,  and  government  and 
non-government  agencies.  This  representation  of  the  OODA  cycle  highlights  the  need  to  align 
interdependent  visions  and  to  coordinate  multi-player  action  cycles. 


Cblrhli&jT* 
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Figure  2.  Overlapping  friendly  and  enemy  OODA  cycles  with  external  players. 

Anchored  to  the  OODA  cycle,  a  set  of  eight  visualization  skills  were  identified  that  serve  as 
END  STATE'S  terminal  learning  objectives.  The  first  two  visualization  skills,  for  example,  relate  to  the 
Observe  segment  of  OODA.  Users  of  END  STATE  initially  learn  to  observe  and  envision  how  the 
space  and  time  factors  of  their  Area  of  Operations  (AO)  might  impact  maneuver,  action,  and 
coordination.  Next,  users  observe  and  envision  how  their  AO  might  impact  Mission,  Enemy,  Terrain  & 
Weather,  Troops  &  Support,  Time  Available,  and  Civilians  (METT-TC)  factors  as  they  refine  their 
visualization  to  match  their  unit’s  mission,  task,  and  purpose.  The  complete  set  of  visualization  skills 
addressed  by  END  STATE  is  as  follows: 
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1 .  Observe  and  envision  the  AO  in  2D,  3D  and  4D. 

2.  Observe  and  envision  the  AO’s  METT-TC  factors. 

3.  Orient  the  unit  for  maneuver,  action,  and  coordination. 

4.  Orient  external  players  for  maneuver,  action,  and  coordination. 

5.  Decide  what  information  is  needed  for  current  and  future  action. 

6.  Decide  what  actions  are  needed  for  current  and  future  situations. 

7.  Act  on  emerging  threats. 

8.  Act  on  emerging  opportunities. 

Each  visualization  skill  is  associated  with  multiple  performance  criteria,  which  provide  targets 
for  learner  interaction  with  the  training  program  and  performance  assessment.  The  complete  set  of 
performance  criteria  for  the  eight  visualization  skills  is  provided  by  Lickteig,  et  al.  (2009).  As  an 
example,  the  performance  criteria  for  the  first  visualization  skill,  Observe  and  Envision  the  AO  in  2 
Dimensional  (2D),  3  Dimensional  (3D),  and  4  Dimensional  (4D),  are  provided  below: 

■  Estimate  the  relative  distance  and  time  required  by  troops,  assets,  and  the  enemy  to  travel  from 
one  point  to  another  in  the  operational  environment  at  different  times  of  day,  days  of  the  week, 
points  in  the  year,  under  different  weather  conditions,  and  using  different  modes  of  movement. 

■  Comprehend  the  implications  of  physical  terrain,  including  terrain  elevation,  urban  elevation, 
and  line-of-sight. 

■  Judge  the  orientation,  dispersion,  and  visibility  of  target  points  in  the  field  from  reference  points 
in  the  field. 

■  Comprehend  the  implications  of  human  terrain  for  friendly  and  enemy  action. 

END  STATE  Overview 

This  section  provides  a  brief  overview  of  the  current  END  STATE  training  product,  particularly 
the  product  refinements  made  to  meet  the  research  objectives  previously  listed.  A  more  complete 
description  of  END  STATE’S  design,  development,  and  formative  evaluation  is  available  (Lickteig,  et 
al.,  2009)  as  well  as  the  current  END  STATE  training  product  (ARI,  in  preparation). 

Soldier  Summary 

After  initially  starting  END  STATE  and  creating  a  new  user  profile,  the  user  is  presented  the 
Soldier  Summary  depicted  in  Figure  3.  The  Soldier  Summary  provides  a  brief  description  of  END 
STATE  and  its  goals:  training  junior  leaders  to  visualize  small  unit  operations  and  coaching  them  to 
apply  those  skills  on  a  full-spectrum  battlefield.  The  summary  stresses  that  END  STATE  is  a  tool 
designed  to  guide  professional  development  and  discussion  that  can  be  used  in  either  institutional  or 
unit  settings.  The  summary  also  informs  the  user  that  although  END  STATE  is  set  in  Kirkuk,  Iraq,  the 
training’s  principles  on  how  to  visualize  and  conduct  small  unit  operations  apply  in  any  COIN 
environment. 
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COMMANDER’S  VISUALIZATION  AT  COMPANY  LEVEL 


SOLDIER  SUMMARY  FOR  ENDSTATE 


ENDSTATE  is  a  multimedia,  interactive  training  program  designed  to 
accomplish  two  primary  goals.  The  first  is  to  train  junior  leaders  in  the 
skills  of  battlefield  visualization;  the  second,  to  coach  the  same  leaders 
in  applying  those  skills  on  a  full-spectrum  battlefield.  This  training  can 
be  employed  either  in  an  institutional  or  unit  environment. 

ENDSTATE  is  designed  to  stimulate  and  guide  leader  development  and 
professional  discussion;  it  is  not  intended  to  provide  “school-house 
solutions”  to  unpredictable  and  complex  operations.  However,  it  can 
never  replace  the  interaction  that  should  occur  between  experienced 
leaders  and  inexperienced  ones. 

ENDSTATE  is  set  in  Kirkuk,  Iraq;  but  its  principles  apply  anywhere. 
Users  of  ENDSTATE  will  learn  to  apply  the  principles  of 
counterinsurgency  that  are  expressed  in  FM  3-24  as  the  tactical  defeat 
mechanism.  These  principles  apply  to  any  counterinsurgency;  they  are 
not  unique  to  time  or  place. 


CONTINUE 


Figure  3.  Soldier  Summary. 

Course  Map 

After  a  user  selects  the  Continue  button  on  the  Soldier  Summary  page,  the  Course  Map  depicted 
in  Figure  4  is  presented.  END  STATE'S  Course  Map  is  the  centralized  location  where  users  can  access 
all  elements  of  the  training  product  including  the  training  vignettes,  the  tests,  and  the  training  support 
materials.  Ideally,  the  user  begins  by  reviewing  the  Quick  Start  Guide,  located  at  the  left  side  of  the 
Course  Map.  The  Quick  Start  Guide  notes  the  lock-step  nature  of  END  STATE  which  requires  that 
individuals  with  “user”  profiles  complete  the  16  vignettes  in  order:  the  pre-test  vignette,  the  14  training 
vignettes,  and  then  the  post-test  vignette.  Notably,  END  STATE  now  includes  a  “trainer”  profile  that 
unlocks  the  vignettes  and  allows  small-group  trainers  to  select  any  vignette  or  visualization  skill,  as 
desired,  in  order  to  stimulate  and  guide  professional  discussion  and  development. 

Access  to  the  training  support  materials,  including  tactical  and  instructional  information,  is 
available  via  the  “tool  bar”  across  the  top  of  the  Course  Map.  Clicking  on  the  name  of  each  support 
resource,  such  as  the  Introduction  or  the  Road  to  War,  takes  the  learner  directly  to  the  requested  training 
support  material.  “Blue”  navigational  prompts  are  provided  to  aid  progression  on  the  Course  Map  and 
throughout  the  course.  When  the  learner  begins  the  course,  for  example,  a  blue  highlight  surrounds  the 
pre-test  titled  “Attend  AAR”  module  located  directly  below  the  top  toolbar. 
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First  Name:  William 
Last  Name:  Shakespeare 
Rank: 


Quick  Start  Guide  13 


This  Company  Commander’s 
Visualization  course  is  a  series  of 
16  modules.  Completing  each 
module  requires  about  30  minutBS 
and  unlocks  the  next. 

1.  Begin  with  Attend  AAR  to  pre¬ 
test  your  visualization  skills. 

2.  Next,  complete  the  training 
modules  in  order.  Each  “a’ 
module  helps  PREPARE  you  to 
EXECUTE  the  mission  in  the 
“b”  module. 

3.  End  with  Conduct  AAR  to  post¬ 
test  your  improved  visualization 


Tactical  information  is  at  the  tabs 
Road  to  War,  ROE,  and  BUB. 

Instructional  information  like 
course  structure  and  learning 
objectives  is  at  the  tab 
Introduction. 

Navigation  prompts  are  provided 
in  blue  font  and  highlights  (e.g.. 
see  Attend  AAR). 
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Attend  AAR 


OBSERVE 


3a  Seize  Sawha 


ORIENT 

'l 


3b  Seize  Sawha 


2a  Escort  Convoy 


Training 

Testing 

■ 

'onduct  AAR 

1 

Status  Trained  = 

Prepare  =  *aT  (1  a-7 a) 
Execute  =  “b*  (1b-7b) 

Pre-test  =  Attend  AAR 

Post-test  =  Conduct  AAR 

Click  If  for  Your  Feedback 

Practice  Needed  - 

Untrained  = 

Figure  4.  END  STATE’S  Course  Map. 

As  the  user  progresses  through  END  STATE,  the  navigational  prompts  advance  to  the  next  step. 
For  example  in  Figure  4,  the  notional  user  has  completed  the  pre-test  and  the  first  six  training  vignettes, 
so  a  blue  highlight  (in  color  versions  of  the  figure)  surrounds  the  next  training  vignette  titled  “4a 
Extend  Presence”  under  the  Orient  segment  of  the  OODA  cycle.  As  the  user  completes  each  vignette, 
its  padlock  icon  is  replaced  by  an  information  icon  by  which  users  can  view  feedback  on  their 
responses  to  each  test  item  and  training  thought  prompt. 

The  legend  at  the  lower  left  of  the  Course  Map  provides  a  course  synopsis  on  how  the  training 
is  structured  into  pairs  of  “a”  and  “b”  vignettes  which  are  labeled  Prepare  and  Execute,  respectively. 
The  Quick  Start  Guide  reinforces  the  training  program’s  reflection-action  cycle  by  stating:  “Each  ‘a’ 
module  helps  PREPARE  you  to  EXECUTE  the  mission  in  the  ‘b’  module.”  The  legend  at  the  lower 
right  of  the  Course  Map  shows  status  indicators  of  green  (in  color  versions)  for  “Trained,”  amber  for 
“Practice  Needed,”  and  red  for  “Untrained.”  After  each  vignette  is  completed,  the  Course  Map  updates 
and  displays  the  user’s  current  status  on  all  completed  vignettes. 

Training  Support  Materials 

Quick  Start  Guide.  As  described  previously,  the  Quick  Start  Guide  provides  a  brief  set  of 
instructions  on  how  to  use  the  training  system.  It  appears  as  shown  in  Figure  4  (above)  when  the 
learner  visits  the  Course  Map  for  the  first  time.  On  subsequent  visits,  the  Quick  Start  Guide  is  not 
shown,  but  may  be  accessed  by  clicking  the  Quick  Start  Guide  button. 
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Course  Introduction.  The  Course  Introduction  provides  a  brief  self-paced,  illustrated  text 
overview  of  the  components  of  the  END  STATE  program.  This  overview  also  includes  frequently 
asked  questions  about  how  the  system  functions. 

Road  to  War.  The  Road  to  War  is  a  self-paced  multimedia  presentation  of  country-level 
background  information  on  Iraq,  the  setting  of  each  of  the  END  STATE  vignettes.  The  presentation 
includes  information  on  Iraq’s  population,  economy,  and  culture  as  well  as  a  high-level  description  of 
recent  political  events  in  the  area. 

Rules  of  Engagement  (ROE).  The  ROE  are  a  text  presentation  of  10  notional  rules  of 
engagement  for  conducting  operations  in  Iraq.  The  ROE  have  been  enhanced  to  reflect  the  realities  of  a 
sovereign  Iraq.  Specifically,  they  contain  rules  for  “sharing  the  road”  with  Iraqis  and  for  warrant-based 
detention  operations. 

Battle  Update  Briefs.  END  STATE  features  two  multimedia  update  briefs:  one  at  the  battalion 
level  and  one  at  the  company  level.  Both  briefs  use  a  combination  of  still  images  and  3D-animated 
presenters  to  convey  the  information  in  an  engaging  and  realistic  fashion.  Information  in  each  of  the 
briefs  includes  intelligence  on  people,  places,  and  tactics  at  an  echelon-specific  level  of  analysis. 

Training  Vignettes  and  Thought  Prompts 

The  storyline  of  END  STATE  features  a  3D  avatar  called  Captain  Dukes  of  A/2-5  IN  conducting 
full-spectrum  operations  with  his  unit  in  Kirkuk,  Iraq.  Beginning  with  his  unit’s  relief-in-place  during 
the  first  four  training  vignettes,  Captain  Dukes  learns  to  carefully  observe  his  AO  and  especially  the 
area’s  human  and  physical  terrain  and  the  enemy’s  deceptive  patterns  of  operation.  During  the  next 
four  training  vignettes,  Dukes  repeatedly  orients  his  unit  and  key  external  players  for  coordinated 
action  by  sharing  his  understanding  of  the  current  situation  and  its  desired  endstate.  The  next  four 
vignettes  examine  how  Dukes  decides  what  actions  his  unit  will  need  to  take  to  achieve  the  desired 
endstate.  In  the  last  two  training  events,  Dukes  and  his  unit  repeatedly  act  to  exploit  emerging 
opportunities  and  deter  emerging  threats  in  their  AO.  The  14  training  vignettes  are  organized  into  seven 
complementary  pairs,  as  shown  in  the  Course  Map,  in  which  Dukes  considers  relevant  principles  for 
visualizing  and  conducting  the  upcoming  mission  and  then  applies  the  principles  during  execution  of 
the  mission. 

Throughout  the  training  and  testing  vignettes,  a  3D  avatar  called  Major  Harris  serves  as  Captain 
Dukes’  virtual  mentor.  Major  Harris,  Dukes’  former  small-group  instructor  in  the  Captains’  Career 
Course,  repeatedly  appears  during  internal  dialogues  in  which  Harris  guides  Dukes’  thoughts  and 
decision-making.  Major  Harris  also  presents  the  74  thought  prompts  that  challenge  Dukes  and  the 
learner  to  apply  their  visualization  skills  to  each  mission  and  then  receive  Harris’  explanatory  feedback 
in  the  form  of  expert  considerations  to  all  user  responses.  Coupling  the  thought  prompts  with 
explanatory  feedback  is  designed  to  stimulate  and  reinforce  the  reflective  thought  and  deliberate 
practice  needed  to  ingrain  visualization  principles  and  skills.  Figure  5  shows  an  example  thought 
prompt  with  explanatory  feedback. 
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Figure  5.  Sample  feedback  to  a  learner  on  a  thought  prompt. 

Each  user  is  given  three  attempts,  if  needed,  to  achieve  100%  correct  on  each  thought  prompt. 
After  each  attempt  at  least  partial  feedback  is  provided,  see  Figure  5  for  Harris’  feedback  on  the  three 
answers  marked  as  correct.  Although  data  on  all  attempts  are  captured  for  trainers  and  researchers,  only 
data  from  the  last  attempt  are  scored  for  the  results  feedback  that  learners  receive.  Learners  get  partial 
credit  for  partially  correct  solutions  on  all  thought  prompts,  except  for  multiple-choice  items.  These 
design  features  for  thought  prompts  stress  END  STATE’S  emphasis  on  motivating  and  reinforcing 
learning.  For  example,  select-all-that  apply  thought  prompts  are  scored  by  summing  the  proportional 
score  associated  with  each  correct  selection  and  each  correct  rejection.  If  the  user  must  select  three  of 
five  response  options;  the  user  receives  20%  (1/5)  for  each  option  correctly  selected  and  each  option 
correctly  rejected.  Such  scoring  prevents  users  from  scoring  100%  simply  by  selecting  all  response 
options  regardless  of  their  applicability. 

As  thought  prompts  and  expert  feedback  were  considered  instrumental  to  learning,  their 
revision  was  the  primary  concern  during  the  current  research  effort.  Revisions  to  the  thought  prompts 
and  the  explanatory  feedback  provided  by  Harris  highlight  the  need  to  adapt  solutions  to  the 
complexity  and  uncertainty  of  COIN  (DA,  2006).  By  design,  the  thought  prompts  will  stimulate 
introspection  by  junior  leaders  working  as  individual  users  and  facilitate  discussion  and  professional 
development  in  small-group  settings.  Overall,  the  thought  prompts  stress  the  need  to  proactively  think 
through  decisions  before  acting — the  cruxes  of  the  OODA  cycle — the  challenge  to  small  unit 
operations  in  COIN. 
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Pre-  and  Post-Tests 


The  pre-  and  post-tests  were  developed  to  assess  the  value  of  END  STATE’S  training.  In 
addition,  the  tests  may  increase  learners’  motivation  to  improve  their  visualization  skills  when,  for 
example,  they  discover  their  pre-test  performance  is  inferior  to  that  of  experienced  junior  leaders. 

The  storyline  of  the  pre-  and  post-tests  extends  the  story  of  Captain  Dukes  to  include  the 
incoming  and  outgoing  After  Action  Reviews  (AARs)  for  his  unit’s  deployment  in  Kirkuk,  Iraq.  Test 
items  are  embedded  in  AAR  discussions.  For  the  pre-test,  the  discussion  is  led  by  the  outgoing 
company  commander  that  Captain  Dukes  is  replacing;  for  the  post-test,  the  discussion  is  led  by  Dukes 
for  his  incoming  replacement  (see  Attend  AAR  and  Conduct  AAR  in  Figure  4).  Each  test  comprises  16 
questions,  structured  by  the  visualization  skills  required  to  See  First,  Understand  First,  and  Act  First 
and  their  direct  relation  to  the  OODA  cycle  (DA,  2003).  The  AARs  are  conducted  in  the  3D  world  of 
END  STATE  with  Major  Harris  introducing  each  test  item,  as  appropriate  to  the  ongoing  AAR 
discussion.  In  contrast  to  the  thought  prompts  during  training,  learners  are  given  only  one  chance  to 
answer  each  test  item  and  do  not  receive  immediate  feedback  after  submitting  their  response  to  a  test 
item.  However,  test  performance  summaries  and  item-level  results  are  available  to  learners  after  the 
completion  of  each  test  via  the  information  icons  shown  on  the  Course  Map  for  each  test. 

The  pre-  and  post-tests  now  included  in  END  STATE  are  the  result  of  extensive  research, 
testing,  and  refinement  in  collaboration  with  more  than  200  Soldiers,  including  nearly  80  for  test 
revision  and  more  than  150  for  test  evaluation.  The  results  of  that  research  are  briefly  summarized 
below;  more  detailed  documentation  of  the  pre-  and  post-test  revision  and  evaluation  are  provided  in 
Appendix  B.  In  the  test  revision  phase,  all  test  items  were  individually  completed  and  reviewed  by 
nearly  80  junior  leaders  with  deployment  experience.  Based  on  Soldier  reviews  of  what  were  labeled 
Test  A  and  Test  B  forms,  many  of  the  test  items  were  revised  by  the  research  team  led  by  a  co-author 
recently  deployed  to  Iraq  and  now  a  retired  lieutenant  colonel.  Item  modifications  ranged  from 
relatively  minor  changes  (e.g.,  clarification  of  question  stems),  to  more  accurate  and  current  stimulus 
materials  and  terminology,  to  entire  replacement  of  item  content.  In  the  test  evaluation  phase,  the 
revised  versions  of  the  Test  A  and  Test  B  forms  were  administered  to  more  than  150  junior  leaders 
including  cadets,  captains,  lieutenants,  and  senior  NCOs.  After  filling  out  a  brief  demographic  survey, 
the  Soldier  participants  completed  both  test  forms  in  paper-and-pencil  format.  The  order  of  test  form 
administration  was  counterbalanced  across  Soldier  participants.  In  the  analysis  phase,  the  resulting  test 
scores  were  examined  for  differences  between  test  forms  and  for  differences  between  experienced  and 
inexperienced  junior  leaders,  operationally  defined  as  with,  versus  without,  deployment  experience, 
respectively. 

The  revised  test  forms  were  found  to  be  sufficiently  comparable  to  enable  training  effectiveness 
evaluation  with  overall  average  means  of  67%  and  69%  for  the  pre-  and  post-test,  respectively.  The 
corrected  correlation  between  the  two  test  forms  was  .85,  and  differences  in  average  scores  across  the 
two  test  forms  were  statistically  non-significant. 

The  revised  test  forms  were  found  to  discriminate  between  junior  leaders  with  and  without 
deployment  experience,  as  shown  in  Figure  6.  Performance  differences  between  inexperienced  and 
experienced  leaders  on  both  test  forms  were  significantly  different,  with  large  effect  sizes  (>1). 
Statistical  results  are  provided  in  Appendix  B. 
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%  Correct 


Figure  6.  Test  performance  by  test  form  and  junior  leader  experience. 

Training  Data  Presentation 


Learner  Data 

As  noted,  an  objective  of  the  research  was  to  improve  access  to,  and  the  utility  of,  learner 
performance  data.  Revisions  to  END  STATE  now  enable  learners  to  more  readily  review  their  status  on 
each  test  and  training  vignette.  The  purpose  of  these  updates  was  to  provide  learners  with  the 
opportunity  to  more  readily  and  precisely  track  their  training  and  test  status  in  detail  and  to  maximize 
the  learning  benefit  of  reflective  thought  and  deliberate  action  with  repeated  access  to  expert 
considerations  and  explanations. 

At  the  completion  of  any  vignette,  the  learner  is  automatically  returned  to  the  Course  Map 
where  the  learner’s  status  color  for  the  vignette  is  updated  (green,  amber,  or  red1)  and  its  padlock  icon 
is  replaced  with  an  information  icon  (see  Figure  4).  After  clicking  on  an  information  icon,  learners  are 
presented  their  overall  score  for  that  vignette  as  well  as  color-coded  status  feedback  on  their 
performance  for  each  test  item  or  thought  prompt  in  the  vignette,  as  shown  in  Figure  7. 

For  additional  feedback  on  thought  prompts,  learners  can  click  any  “View”  button  on  the  right 
side  of  Figure  8  to  review  the  Answer  Key  for  any  thought  prompt.  The  Answer  Key  presents  the 
thought  prompt  as  it  appeared  during  training  with  the  addition  of  correct  and  incorrect  response 
indicators  and  expert  explanatory  feedback.  After  reviewing  the  feedback,  learners  can  go  back  and  re¬ 
take  vignettes  to  enhance  their  training  status.  For  the  pre-  and  post-test  vignettes  (Attend  AAR  and 
Conduct  AAR)  feedback  options  and  formats  are  similar,  however,  no  Answer  Key  is  provided. 


1  green  =  85%+  correct;  amber  =  70-84%  correct;  red  =  less  than  70%  correct 
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0Back  Vignette:  2b  Escort  Convoy 

Your  Overall  Vignette  Score  =  70 


COL  U  Check  on  Learning  (COL) 

22  Select  the  two  most  likely  obstacles  in  this  urban  scene. 


23  What  is  the  best  action  when  reacting  to  this 


24  Match  each  action  with  its  two  most  likely  results. 


25  |  Select  the  two  most  likely  obstacles  in  this  scene. 

View  | 

26  What  five  results  does  the  enemy  desire  from  a  friendly  convoy  encountering  an  IED? 

mam 

27  Rank  these  actions  from  best  to  worst  to  assist  Mr.  Smith . 

28  Select  the  key  terrain  along  this  route  from  the  areas  identified. 

Click  ^EBB  to  View  Answer  Key. 
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Figure  7.  Sample  feedback  to  learner  on  a  completed  training  vignette. 
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(7)  Back 


Rank  these  actions  from  best  to  worst  to  assist  Mr.  Smith. 


Drag  the  letter  for  each  option  to  the  correct  location. 


Setup  a  perimeter  and  post  guards 
until  Smith’s  security  force. 


B  Ensure  that  Smith  has  a  way  to 
contact  you  and  head  back  to  the 

C  Walk  the  perimeter  with  Smith  and 
move  his  equipment  into  a 

Leave  a  couple  of  soldiers  to  guard 
the  power  station  and  parts. 


A_ 

/-C_ 

/-§. 


(Best) 


This  may  be  the  best  option,  however  you  are  not  prepared  to 
conduct  this  for  an  extended  time. 


This  option  will  at  least  help  Smith  make  his  site  less  vulnerable. 


Smith  can  call  if  things  turn  ugly  and  you  can  respond. 


Leaving  two  soldiers  to  guard  a  site  would  be  asking  for  an  insurgent 
attack. 


Change  you  patrol  schedule  to  1  _  Don’t  compromise  your  entire  mission.  If  you  cannot  handle  this  new 

commit  all  your  forces  to  guarding  V  — (Worst)  requirement,  ask  battalion  for  help. 


Back 


Figure  8.  Sample  answer  key  for  a  thought  prompt. 
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Trainer  and  Researcher  Data 


For  trainers  and  researchers,  more  quantitative  data  are  captured  by  END  STATE  and  stored  in  a 
format  that  can  be  readily  imported  into  most  spreadsheet  applications  (e.g.,  Microsoft  Excel).  In  the 
present  effort,  updates  were  made  to  data  collection  and  storage  in  order  to  make  it  easier  for 
researchers  to  analyze  the  data  in  a  variety  of  ways.  Specifically,  END  STATE  records  the  following 
data  elements  based  on  learner  performance,  each  in  a  separate  data  field: 

•  A  unique  user  identification  number. 

•  The  user’s  initial  score  on  each  thought  prompt  or  test  question. 

•  The  user’s  final  score  on  each  thought  prompt  (not  applicable  to  test  questions). 

•  The  number  of  attempts  made  to  answer  the  question  correctly  (not  applicable  to  test  questions). 

•  Date  and  time  the  data  were  submitted. 

Product  Transition  Status 

Visualization  training  must  be  available  anytime  and  anywhere  in  order  to  support  the 
demanding  schedules  of  today's  junior  leaders.  For  this  reason,  END  STATE  was  implemented  as  a 
client-side  application  that  can  run  anywhere  a  learner  has  computer  access.  Installation  is  a  simple 
matter  of  running  a  DVD,  and  program  use  requires  no  other  special  applications  or  plug-ins. 

The  current  research  included  efforts  to  enable  the  storage  of  training  data  in  a  central 
repository  on  an  Army  server.  Centralized  data  storage  would  enable  Army-wide  training  effectiveness 
evaluation.  Although  learners  can  run  the  END  STATE  application  on  any  computer,  Internet 
connection  and  common  access  card  (CAC)  authentication  is  required  to  submit  data  to  this  central 
repository. 


Recommended  Future  Directions 

During  earlier  development  research  and  formative  evaluation,  nearly  50  experienced  junior 
leaders  concluded  that  the  END  STATE  was  relevant,  effective,  and  worth  using  (Lickteig,  et  al.,  2009). 
In  the  current  research  reported  here,  data  from  more  than  150  Soldiers  indicated  that  END  STATE’S 
pre-  and  post-training  tests  are  comparable  and  able  to  identify  significant  differences  in  the 
visualization  skills  of  inexperienced  versus  experienced  junior  leaders. 

The  natural  next  step  in  END  STATE’S  transition  is  a  comprehensive  training  effectiveness 
evaluation  that  assesses  changes  in  knowledge  and  skill  as  a  result  of  exposure  to  the  learning  content 
(Kirkpatrick,  1994).  Such  an  evaluation  would  further  strengthen  the  product  by  revealing  areas  where 
refinement  could  improve  achievement  of  the  visualization  learning  objectives.  Advanced  training 
effectiveness  evaluation  could  then  address  behavior  changes  in  the  operational  environment  and 
broader  organizational  impact. 

Training  effectiveness  evaluation  might  begin  by  determining  if  there  are  significant  differences 
in  the  pre-  and  post-test  performance  of  junior  leaders  after  completing  END  STATE’S  approximately 
6.5  hours  of  training.  Evaluation  might  also  compare  performance  differences  on  the  post-test  between 
learners  who  completed  the  training  vignettes  and  those  who  did  not.  Ideally,  participants  in  the 
evaluation  should  be  representative  junior  leaders  without  deployment  experience  to  ensure 
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generalizable  results.  Future  evaluations  should  assess  the  training  effectiveness  of  END  STATE  on 
job-related  performance  metrics. 

Evaluating  training  effectiveness  using  segments  of  the  training  could  reveal  the  minimum 
number  of  vignettes  necessary  to  achieve  a  significant  impact  on  visualization  knowledge  and  skill.  It 
would  also  provide  data  that  leaders  could  use  to  judge  their  return  on  investment  in  the  END  STATE 
training.  That  is,  the  data  could  show  how  much  improvement  in  visualization  knowledge  and  skill  the 
average  leader  could  expect  after  investing  a  certain  number  of  hours  in  the  training.  The  data  might 
also  be  useful  to  small-group  trainers  for  selecting  the  most  useful  vignettes  and  visualization  training, 
as  needed,  to  stimulate  and  guide  professional  discussion  and  development. 

Conclusions 

END  STATE’S  potential  training  audiences  include  all  junior  leaders,  both  officers  and  NCOs, 
across  a  broad  range  of  training  venues.  The  product’s  DVD  format  readily  accommodates  training 
delivery  options  ranging  from  a  lieutenant  with  a  laptop  to  a  group  in  a  battalion-level  NCOPD  session. 
END  STATE  is  a  relevant,  flexible,  and  effective  tool  that  will  enable  junior  leaders  to  develop 
visualization  skills  through  their  own  effort,  not  at  the  expense  of  Soldiers  on  the  ground.  By  preparing 
young  leaders  to  visualize  the  challenges  of  COIN  using  virtual  training  like  END  STATE,  the  U.S. 
Army  can  maximize  their  success. 
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Appendix  A 


Acronyms 


2D 

2  Dimensional 

3D 

3  Dimensional 

4D 

4  Dimensional 

AAR 

After  Action  Review 

AO 

Area  of  Operations 

ARI 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

CAC 

Common  Access  Card 

COIN 

Counterinsurgency 

DA 

U.S.  Department  of  the  Army 

METT-TC 

Mission,  Enemy,  Terrain  &  Weather,  Troops  &  Support,  Time  Available, 
and  Civilians 

NCO 

Non-commissioned  Officer 

NCOPD 

NCO  Professional  Development 

OODA 

Observe,  Orient,  Decide,  Act 

ROE 

Rules  of  Engagement 

A-l 
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Appendix  B 

Pre-  and  Post-Test  Refinement  and  Evaluation 
Initial  Test  Form  Revisions 

Refinement  of  the  pre-and  post- tests  began  with  comprehensive  subject  matter  expert  review 
and  modifications  to  the  original  test  forms  described  in  Lickteig,  et  al.  (2009).  Modifications 
addressed  comments  made  during  the  user  review  process  and  reviews  made  by  the  independent 
consultant  on  the  current  effort.  Revisions  ranged  from  changes  to  relatively  minor  elements  of  the 
assessment  items  (e.g.,  clarification  of  question  stems)  to  updates  of  visuals  and  terminology  to  entire 
replacement  of  item  content.  Although  individual  assessment  items  were  revised,  sometimes 
substantially,  the  overarching  format  (“Army  Qualities  of  Firsts”)  and  storyline  (Attend/Conduct  an 
AAR)  of  the  original  test  forms  were  retained.  Modified  test  forms  were  then  subjected  to  evaluation 
by  a  new  sample  of  military  subject  matter  experts  located  at  Fort  Campbell  (AT  =31)  who  were 
instructed  to  complete  one  of  the  two  test  forms,  then  provide  evaluative  ratings  for  each  assessment 
item.  A  copy  of  the  evaluation  form  is  provided  in  Appendix  C.  The  test  forms  were  revised  based  on 
the  data  from  Fort  Campbell,  and  then  subjected  to  further  analysis  and  revision  as  described  below. 


Participants 

One  hundred  fifty-nine  participants  from  two  different  installations  provided  test  form 
performance  data.  Thirty-eight  percent  (N=  60)  of  the  participants  were  commissioned  officers  (72% 
were  captains,  27%  were  lieutenants  and  there  was  one  major),  4%  (N  =  6)  were  non-commissioned 
officers  (all  but  one  were  sergeants  first  class),  and  the  remaining  58%  were  cadets.  The  highest  level 
of  military  education  achieved  by  43%  of  the  commissioned  officers  (N=  26)  was  the  Officer  Basic 
Course,  with  the  remaining  officers  having  completed  the  Officer  Advanced  Course  (N=  33)  or  the 
Pre-Command  Course  (N=  1).  Of  the  six  enlisted  Soldiers,  two  reported  having  completed  the  Basic 
Non-Commissioned  Officers’  Course,  three  reported  having  completed  the  advanced  course,  and  one 
participant  reported  having  not  yet  completed  any  courses  in  the  non-commissioned  officer  education 
system  (this  participant  was  a  specialist,  E-4).  Eighty-three  percent  of  commissioned  officers  reported 
having  command  experience.  Thirty-eight  percent  of  participants  reported  having  done  a  tour  in 
Afghanistan  or  Iraq,  with  20  participants  having  served  on  multiple  tours  (2-4  tours).  Two  participants 
were  removed  from  analyses  for  having  not  completed  both  the  pre-  and  post-tests,  reducing  the  sample 
size  to  157. 


Materials 

To  enable  rapid,  flexible  administration  of  the  training  assessments  in  the  context  of  minor 
content  and  format  changes,  paper-and-pencil  format  was  used.  Tike  the  original  test  forms,  those 
forms  used  for  final  evaluation  each  contained  16  items  that  assessed  the  eight  visualization  skills 
taught  in  END  STATE.  Also  consistent  with  the  original  test  forms,  the  format  of  the  assessment  items 
included  multiple  choice,  matching,  and  select  all  that  apply.  Participants  were  given  only  one  chance 
to  answer  each  item  and  overall  performance  scores  were  calculated  as  the  average  percent  correct  for 
each  item.  Every  effort  was  made  to  balance  the  number  of  multiple-choice  items  across  the  two  test 
forms  in  order  to  standardize  the  expected  performance  scores,  however,  there  were  slightly  more 
multiple  choice  items  in  the  post-test.  Given  that  some  revisions  were  made  to  the  test  forms  during  the 


B-l 


data  collection  process,  not  all  participants  received  exactly  the  same  test  form.  In  the  pre-test,  13  of  the 
16  test  questions  were  exactly  the  same  or  functionally  equivalent  across  all  participants.  In  the  post¬ 
test,  9  of  the  16  questions  were  exactly  the  same  or  functionally  equivalent. 

Procedure 

All  participants  began  by  filling  out  a  brief  demographic  survey.  Next,  they  were  administered 
both  test  forms  back-to-back  via  a  counterbalanced  research  design  (i.e.,  half  received  the  pre-test  first 
and  half  received  the  post-test  first).  Twenty-five  participants  evaluated  the  pre-test  and  eighteen 
evaluated  the  post-test  using  the  evaluation  form  shown  in  Appendix  C.  None  of  the  participants 
received  any  END  STATE  training. 


Results  and  Discussion 

Initial  analyses  were  conducted  to  determine  whether  the  final  form  of  the  pre-  and  post-tests 
(containing  all  16  items  across  participants)  could  be  considered  functionally  equivalent  to  its 
corresponding  “abbreviated”  form  (containing  only  those  items  that  were  the  exactly  the  same  or 
trivially  different  across  participants).  Functional  equivalence  must  be  demonstrated  in  order  to 
investigate  the  adequacy  of  the  final  test  forms  for  assessing  pre-  and  post-training  performance.  The 
correlations  between  the  abbreviated  and  final  test  forms  were  found  to  be  substantial  (.95  for  the  pre¬ 
test  and  .89  for  the  post-test),  indicating  that  the  final  test  forms  measured  the  same  (or  very  highly 
similar)  constructs  as  their  corresponding  abbreviated  forms,  despite  differences  in  some  test  questions. 
Moreover,  the  average  overall  scores  across  final  and  abbreviated  test  forms,  although  statistically 
significantly  different,  were  not  practically  different;  the  effect  sizes  of  the  differences  were  very  small 
and  overall  average  scores  differed  by  only  one  percentage  point.  The  average  scores  on  the  pre-test 
were  66%  and  67%  for  the  abbreviated  and  final  forms,  respectively  [t(156)  =  2.045 ,p  =  .04,  d  =  .05], 
Similarly,  the  average  scores  on  the  post-test  were  70%  and  69%  for  the  abbreviated  and  final  forms, 
respectively  [t(156)  =  -3.155 ,p  =  .002,  d  =  .12].  Together,  these  findings  suggested  that  the  final  test 
forms  could  be  used  for  further  analysis  despite  differences  in  exact  format  across  participants. 

Test  Form  Sensitivity 

The  descriptive  statistics  for  the  pre-  and  post-tests  are  shown  in  the  table  below.  As  can  be  seen 
in  the  table,  the  reliability  of  each  assessment  form  is  moderate,  appropriately  reflecting  the 
multifaceted  nature  of  operational  visualization.  Also  shown  in  the  table  are  the  differing  average 
scores  for  participants  who  had  been  deployed  at  least  once  and  those  who  had  never  been  deployed,  t- 
tests  were  conducted  on  the  pre-  and  post-test  scores  to  evaluate  the  significance  of  these  performance 
differences.  For  both  assessment  forms,  these  differences  were  found  to  be  significant  and  effect  sizes 
quite  large  [t(155)  =  6 .66, p  =  0 ,d=  1.14  for  the  pre-test;  t(155)  =  136, p  =  0 ,d=  1.15  for  the  post¬ 
test].  These  results  clearly  indicate  that  the  pre-  and  post-tests  designed  to  assess  visualization  are 
indeed  sensitive  to  differences  in  actual  operational  experience  with  employing  visualization.  It  is 
possible  that  deployment  experience  did  not  produce  perfect  average  scores  on  the  pre-  and  post-tests 
because  the  participants  who  had  been  deployed  were  still  junior  officers  and  members  of  the  END 
STATE  target  training  audience. 
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Table  1 


Descriptive  Statistics  for  Each  Test  Form 


Test  Form 


Scale 

Reliability 


A  .66 

B  .65 


Overall 
(N=  157) 
67  (17) 
69(15) 


Mean  (SD)  %  Correct 
Inexperienced 
(A  =96) 

61  (16) 

63  (14) 


Experienced 
(N=  61) 
77  (12) 

78 (12) 


Test  Form  Comparability 

In  order  to  ensure  that  performance  differences  between  pre-  and  post-testing  reflect  experience 
with  the  training  content,  the  two  test  forms  must  be  comparable  with  regard  to  the  constructs  they 
assess  and  the  average  overall  scores  that  they  produce.  To  investigate  the  comparability  of  the  pre-  and 
post-tests,  their  inter-correlation  was  analyzed,  as  well  as  the  difference  in  their  average  overall  scores. 
The  correlation  between  test  forms  was  .56.  When  corrected  for  attenuation  due  to  the  moderate 
reliabilities  of  the  test  forms,  the  estimated  correlation  was  .85,  suggesting  that  the  pre-  and  post-test 
forms  assessed  similar  underlying  constructs.  A  1-test  indicated  that  the  average  overall  scores  across 
test  forms  did  not  significantly  differ  [1(156)  =  -1.209 ,p  =  .23].  Together,  these  results  suggest  that  for 
the  purposes  of  conducting  training  effectiveness  evaluation  the  test  forms  may  be  considered 
comparable. 

Learner  Ratings 

Using  a  scale  of  1  to  4  (Strongly  Disagree  to  Strongly  Agree,  with  no  “neutral”  or  “no  opinion” 
ratings)  participants  rated  each  test  question  on  the  characteristics  of  confidence  (of  one’s  own 
accuracy),  relevance,  adequacy  (of  information  provided),  clarity  (of  the  test  question  stem  and  of  the 
response  options),  accuracy  (of  content),  and  realism  (see  Appendix  C).  In  addition,  participants  were 
asked  to  rate  each  test  question  for  the  degree  of  expertise  (operationalized  as  educational  level  and 
operational  experience)  required  to  answer  it  correctly.  Ratings  of  the  level  of  expertise  required  served 
as  an  indicator  of  difficulty  of  each  item  that  was  independent  of  actual  performance.  To  analyze 
learner  ratings  on  these  dimensions,  only  questions  from  the  abbreviated  test  forms  were  used.  The 
questions  on  the  final  test  form  that  were  not  administered  across  all  participants  were  not  rated. 
Consistent  with  performance  scores,  however,  it  is  expected  that  the  difference  in  learner  ratings 
between  the  abbreviated  test  forms  and  the  final  test  forms  would  be  trivial. 

Test  form  quality.  An  overall  item  quality  score  was  created  for  each  test  question  by  averaging 
the  learner  ratings  received  on  each  of  the  seven  dimensions  described  above.  Test  form  quality  scores 
were  then  created  for  the  pre-  and  post-tests  by  averaging  the  overall  item  quality  scores.  The  pre-test 
earned  a  quality  score  of  3. 14  (SD  =  .40)  and  the  post-test  earned  a  score  of  3.30  (SD  =  .42),  out  of  a 
maximum  possible  score  of  4.  An  independent  samples  /-test  was  performed  on  these  learner  ratings 
and  was  found  to  be  non-significant  [1(41)  =  -1.271,/?  =  .21].  These  results  indicate  that  on  average 
users  agreed  with  positive  statements  about  the  quality  of  the  test  questions  and  that  the  test  forms  were 
comparable  in  their  perceived  quality. 
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Test  form  difficulty.  Two  overall  difficulty  scores  were  created  for  each  test  form.  The  first  score 
was  created  by  taking  the  average  frequency  of  user  ratings  (across  test  questions)  of  the  amount  of 
deployment  experience  required  to  answer  the  question  correctly  (no  deployment  experience,  one 
deployment,  or  two  or  more  deployments).  The  second  score  was  created  by  taking  the  average 
frequency  of  user  ratings  (across  test  questions)  of  the  amount  of  professional  education  and  leadership 
experience  required  to  answer  the  question  correctly  (ranging  from  cadet  training  and  no  leadership 
experience  to  advanced  coursework  with  company  command  experience). 

On  average,  68%  of  reviewers  of  the  pre-test  indicated  that  no  deployment  experience  was 
required  and  the  remaining  32%  indicated  that  one  deployment  was  required.  On  average,  52%  of  post¬ 
test  reviewers  indicated  that  no  deployment  experience  was  required  and  44%  indicated  that  one 
deployment  was  required. 

On  average,  95%  of  pre-test  reviewers  indicated  that  a  junior  level  of  professional  education 
(i.e.,  the  Advanced  Officers’  Course  or  Advanced  Non-Commissioned  Officers’  Course  or  lower)  and 
leadership  experience  (i.e.,  platoon-level)  were  required  to  answer  the  test  questions  correctly.  On 
average,  98%  of  post-test  reviewers  indicated  that  a  junior  level  of  professional  education  and 
leadership  experience  were  required. 

These  findings  suggest  that  pre-  and  post-test  difficulty  is  well  calibrated  to  the  characteristics 
of  the  END  STATE  target  audience.  They  also  corroborate  the  performance  differences  between 
experienced  and  inexperienced  participants.  That  is,  the  rated  difficulty  level  of  the  pre-  and  post-tests 
should  differentiate  participants  with  and  without  deployment  visualization  experience,  and  this  is 
exactly  what  they  did.  In  addition,  the  test  forms  appear  to  be  comparable  in  difficulty  ratings,  as  well 
as  overall  average  scores. 
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Appendix  C 

Test  Form  Quality  Rating  Survey 

Directions:  For  the  test  item  on  the  left  page,  rate  your  agreement  with  the  statements  below  using  the 
scale  provided.  Place  a  check  in  the  box  that  best  reflects  your  opinion. 


1.  Confidence 


When  I  answered  this  test  item,  I  was 
confident  that  I  answered  correctly. 


2.  Relevance 

3.  Adequacy 

4.  Clarity 


This  test  item  and  problem  set  is  relevant  to 
company  level  visualization. 

This  test  item  and  problem  set  provide 
adequate  information. 

The  wording  of  this  question  is  clear. 


Strongly 

Disagree 


Disagree  Agree 


Strongly 

Agree 


1  2  3  4 


5.  Clarity 


The  wording  of  each  response  choice  is  clear. 


,  .  The  answer  scored  as  “correct”  is  accurate. 

6.  Accuracy 


7.  Realism 


The  visuals  used  in  this  test  item  reflect  the 
operational  environment. 


8.  What  level  of  education/experience  would  a  Soldier  need  to  answer  this  test  item?  Circle  your 
response. 

•  Cadet/officer  candidate/WLC  graduate  -  no  unit  leadership  experience. 

•  BOLC  III/BNCOC  Graduate  -  new  platoon  leader/squad  leader. 

•  CCC/ANCOC  graduate  -  experienced  platoon  leader/platoon  sergeant. 

•  CCC/ANCOC  graduate  -  experienced  company  commander/ 1SG. 

9.  What  is  the  minimum  level  of  operational  experience  required  to  answer  this  test  item?  Circle  your 
response. 

•  Prior  to  first  deployment. 

•  After  first  deployment. 

•  After  two  or  more  deployments. 

10.  How  would  you  improve  this  item?  Please  indicate  here,  or  feel  free  to  mark  suggestions  on  the 
question  and  responses  on  the  left  side  of  this  page.  Your  improvements  might  address  issues  such  as 
the  test  item’s  relevance,  difficulty,  adequacy,  clarity,  accuracy,  and  realism  or  any  other  suggestions 
including  alternate  items  to  replace  this  item. 
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